The following notebook has been created to analyse the results from the XAI questionnaire titled "Survey of the interpretability of decision trees", available at this Github repository. The goal of the questionnaire was to evaluate the interpretability of decision trees.
The notebook is divided into sections, and each of them takes its name from those present in the XAI questionnaire that will analyze this notebook. Each section will explain what information was provided to the survey participants and highlight the results obtained.
We suggest you to use the "Run all" option of the notebook interpreter instead of running a cell at a time. The option is available in the menu bar in Jupyter environment in Run -> Run All Cells.
The data used in this questionnaire is part of the wine dataset provided by UCI. These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines. Below, you can find the first ten rows of the dataset.
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | Class | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 14.23 | 1.71 | 2.43 | 15.6 | 127.0 | 2.80 | 3.06 | 0.28 | 2.29 | 5.64 | 1.04 | 3.92 | 1065.0 | Class 1 |
| 1 | 13.20 | 1.78 | 2.14 | 11.2 | 100.0 | 2.65 | 2.76 | 0.26 | 1.28 | 4.38 | 1.05 | 3.40 | 1050.0 | Class 1 |
| 2 | 13.16 | 2.36 | 2.67 | 18.6 | 101.0 | 2.80 | 3.24 | 0.30 | 2.81 | 5.68 | 1.03 | 3.17 | 1185.0 | Class 1 |
| 3 | 14.37 | 1.95 | 2.50 | 16.8 | 113.0 | 3.85 | 3.49 | 0.24 | 2.18 | 7.80 | 0.86 | 3.45 | 1480.0 | Class 1 |
| 4 | 13.24 | 2.59 | 2.87 | 21.0 | 118.0 | 2.80 | 2.69 | 0.39 | 1.82 | 4.32 | 1.04 | 2.93 | 735.0 | Class 1 |
| 5 | 14.20 | 1.76 | 2.45 | 15.2 | 112.0 | 3.27 | 3.39 | 0.34 | 1.97 | 6.75 | 1.05 | 2.85 | 1450.0 | Class 1 |
| 6 | 14.39 | 1.87 | 2.45 | 14.6 | 96.0 | 2.50 | 2.52 | 0.30 | 1.98 | 5.25 | 1.02 | 3.58 | 1290.0 | Class 1 |
| 7 | 14.06 | 2.15 | 2.61 | 17.6 | 121.0 | 2.60 | 2.51 | 0.31 | 1.25 | 5.05 | 1.06 | 3.58 | 1295.0 | Class 1 |
| 8 | 14.83 | 1.64 | 2.17 | 14.0 | 97.0 | 2.80 | 2.98 | 0.29 | 1.98 | 5.20 | 1.08 | 2.85 | 1045.0 | Class 1 |
| 9 | 13.86 | 1.35 | 2.27 | 16.0 | 98.0 | 2.98 | 3.15 | 0.22 | 1.85 | 7.22 | 1.01 | 3.55 | 1045.0 | Class 1 |
The XAI questionnaire was presented to the evaluation participants with two representations of the same domain of interest. The two representations were in the form of a decision tree, and they differ in dimension/depth. The decision tree representations were:
The following model has a prediction accuracy of 0.927.
The following model has a prediction accuracy of 0.978.
The comprehension test section aims to test the notions acquired by the participants of the questionnaire and to verify the goodness of the participants' mental model. The test is constructed around the sample with the following features.
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 12.42 | 2.55 | 2.27 | 22.0 | 90.0 | 1.68 | 1.84 | 0.66 | 1.42 | 2.7 | 0.86 | 3.3 | 315.0 |
The questionnaire participants have to answer two questions based on the visual explanation that represents the model in the form of a decision tree. The visual explanation (3-Layers Decision Tree) provided is presented below.
The two questions posed to the participants of the questionnaire are the following:
The Comprehension Test section results are:
The "Questions" section of this XAI questionnaire was structured to test the "Transparency" aspect of the explanation provided to the evaluation participants. To assess the "Transparency" aspect of the provided explanations, this section was structured to satisfy the requirements of performing a 'Forward Simulation" task. In Forward Simulation tasks, participants are provided with an input and an explanation to ask them to predict the system's output. The section was composed of five questions, and in each of them, the participants had to answer only one question composed of three choices (the three classes of the dataset). The questions and the explanation in each of them were presented to the participants randomly.
This section will show the prediction accuracy of the participants and the time taken by the participants to answer the questions by taking into account the explanation they received during the evaluation.
NB:
If the data for one of the two explanations would miss, the graph associated with it will not be displayed.
The sample used in question 1 is from the row 0 of the dataset and it has the following features:
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 14.23 | 1.71 | 2.43 | 15.6 | 127.0 | 2.8 | 3.06 | 0.28 | 2.29 | 5.64 | 1.04 | 3.92 | 1065.0 |
The correct classificiation of the sample was class Class 1. Here the results obtained:
The sample used in question 2 is from the row 58 of the dataset and it has the following features:
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 12.37 | 1.13 | 2.16 | 19.0 | 87.0 | 3.5 | 3.1 | 0.19 | 1.87 | 4.45 | 1.22 | 2.87 | 420.0 |
The correct classificiation of the sample was class Class 2. Here the results obtained:
The sample used in question 3 is from the row 156 of the dataset and it has the following features:
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 13.71 | 5.65 | 2.45 | 20.5 | 95.0 | 1.68 | 0.61 | 0.52 | 1.06 | 7.7 | 0.64 | 1.74 | 740.0 |
The correct classificiation of the sample was class Class 3. Here the results obtained:
The sample used in question 4 is from the row 164 of the dataset and it has the following features:
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 13.74 | 1.67 | 2.25 | 16.4 | 118.0 | 2.6 | 2.9 | 0.21 | 1.62 | 5.85 | 0.92 | 3.2 | 1060.0 |
The correct classificiation of the sample was class Class 1. Here the results obtained:
The sample used in question 5 is from the row 177 of the dataset and it has the following features:
| Alcohol | MalicAcid | Ash | AlcalinityOfAsh | Magnesium | TotalPhenols | flavanoids | NonflavanoidsPhenols | Proanthocyanins | ColorIntensity | Hue | OD280-OD315 | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 13.4 | 3.91 | 2.48 | 23.0 | 102.0 | 1.8 | 0.75 | 0.43 | 1.41 | 7.3 | 0.7 | 1.56 | 750.0 |
The correct classificiation of the sample was class Class 3. Here the results obtained:
The questionnaire was completed by 62 participants. Below you will find some information about them.
Participants of the evaluation could select one of the following choices:
Participants of the evaluation could select one of the following choices:
Participants of the evaluation could select one of the following choices:
Participants of the evaluation could select one of the following choices: